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A CLT for an improved subspace estimator with observations of 

increasing dimensions 

P. Vallet*^^t X. Mestre^^t P- Loubaton^^) 


Abstract 

This paper deals with subspace estimation in the small sample size regime, where the number of 
samples is comparable in magnitude with the observation dimension. The traditional estimators, mostly 
based on the sample correlation matrix, are known to perform well as long as the number of available 
samples is much larger than the observation dimension. However, in the small sample size regime, the 
performance degrades. Recently, based on random matrix theory results, a new subspace estimator was 
introduced, which was shown to be consistent in the asymptotic regime where the number of samples and 
the observation dimension converge to inhnity at the same rate. In practice, this estimator outperforms 
the traditional ones even for certain scenarios where the observation dimension is small and of the same 
order of magnitude as the number of samples. In this paper, we address a performance analysis of this 
recent estimator, by proving a central limit theorem in the above asymptotic regime. We propose an 
accurate approximation of the mean square error, which can be evaluated numerically. 
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1 Introduction 

1.1 Motivation 

The problem of subspace estimation, i.e. estimating the eigenspaces of the correlation matrix of a certain 
multivariate time series of dimension M, available from a set of N noisy observations, is an important problem 
in statistical signal processing, and covers several topics such as Direction of Arrival (DoA) estimation 
[14], multiuser detection in Code Division Multiple Access (CDMA) [8], chirp parameter estimation [17] or 
beamforming [3]. Let us consider an complex M-variate time series (yn)n>i, following a ’’signal plus noise” 
model 


Yn — 4” V^, 

where s„ corresponds to a signal part and v„ to a noise part, and assume that N observations yi,... ,yN 
are collected and stacked in the M x N matrix 

Yat = [yi,..., yAf] = Sat + Vtv, 

with Sat = [si,...,SAr] and V^v = [vi, ..., vat]. In many applications, the signals (s„)„>o are moreover 
constrained to a subspace of dimension K less than M and the matrix Sat is full rank K. The subspace 
estimation problem consists in estimating the column space of Sat called the ’’signal subspace”, of dimension 
K (or equivalently its orthogonal complement called the ’’noise subspace” of dimension M — K) from the 
observation matrix Yat. 

The usual way of estimating the signal or noise subspaces consists in estimating their orthogonal pro¬ 
jection matrices. The estimation is performed most of the time by using the so-called sample correlation 
matrix (SCM) of the observations 


N 


1 

N 


N 

n—l 


and these projections are directly estimated by considering their sample estimates, i.e. by considering the 
corresponding orthogonal projection matrix of the SCM. For example, the noise subspace projection matrix 
IIjv, i.e. the orthogonal projection matrix onto the kernel of SpfS*pf, is traditionally estimated by Hat, the 
orthogonal projection matrix onto the eigenspace associated with the M — K smallest eigenvalues of . 

These sample estimators are known to perform well when the number of available samples N is much 
larger than the observation dimension M, in particular because the SCM is a good estimator of the true 
correlation matrix of the observations. Indeed, in the asymptotic regime where M is constant and N 
converges to infinity, under some technical conditions, the law of large numbers ensures that 


Hat — IItv 


0 


almost surely as A —>■ oo, i.e. the sample projection matrices are consistent estimators of the true ones. 
These sample estimators have been also characterized in terms of Central Limit Theorems (CLT) in the 
previous asymptotic regime, and several accurate approximations of the Mean Square Error (MSE) have 
been obtained, see e.g. Anderson [1], Stoica [15], and the references therein. However, it may exist some 
situations where obtaining such an amount of samples is not conceivable, for example in situations where 
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the signals are stationnary only for a short period of time, or simply if the observation dimension is large. 
As a consequence, in the low sample size regime where M and N are of the same order of magnitude, the 
performance of the sample subspace estimators severely degrades, essentially because the SCM does not 
estimate properly the true correlation matrix. 

In this context, based on recent results in random matrix theory, a new subspace estimator was proposed 
by Mestre [11], in the case where (s„)„>o and (v„)„>o are modeled as two independent zero-mean Gaussian 
stationnary processes, temporally uncorrelated, with the signal correlation matrix = E[s„s*] being rank 
K and the noise covariance being equal to cr^I, where ct > 0 and I is the M x M identity matrix, i.e. (yn)n>o 
can be modeled equivalently as 


y„ = (Rs-bcr^l)^^^x„, (1) 

with (x„)„>o a standard spatially and temporally white Gaussian process. Later Vallet et al. [16] obtained, 
using the same approach, a different estimator in the more general situation where the signals (s„)„>o 
are considered as unknown deterministic. The estimators of [11] and [16] were shown to be consistent in 
the asymptotic regime where both the observation dimension M and the number of samples JV converge 
to infinity in such a way that the ratio ^ converge to a positive constant. Moreover, these estimators 
do not assume any particular assumption on the behaviour of the rank K, which may also converge to 
infinity with M, N. In practice, these estimators outperform the traditional ones, when M, N are of the 
same order of magnitude. Based on these results, an application to DoA estimation of K source signals 
impinging on an array of M sensors was proposed, and an improved subspace DoA estimator called G- 
MUSIG (Generalized Multiple Signal Classification) was built, which was shown to numerically outperform 
the traditionnal MUSIC estimator, for realistic values of M, N. This DoA estimator was also shown to be 
consistent in Hachem et al. [7]. 

Recently, Hachem et al. [6] proposed an analysis of the subspace estimator [16], in terms of a Central 
Limit Theorem (CLT), in the previous asymptotic regime where M,N converge to infinity at the same 
rate, and by assuming that the rank K is fixed. In practice, these results are accurate as long as the rank 
K remains small compared to M, N. However, when the rank K is of the same order of magnitude than 
the dimension M and N, the corresponding results do not predict anymore the behaviour of the subspace 
estimator [16] and the results of [6] are not very accurate. 

In this paper, ^ we propose to extend the analysis of [6] regarless the behaviour of the rank K, which 
may increase with M, N. For that purpose, we use a different approach and prove a Central Limit Theorem 
(CLT) in the previous asymptotic regime. We also provide an explicit expression for the Mean Square Error 
(MSE) which can be easily evaluated numerically. Numerical examples confirm the validity of the results. 

The paper is organized as follows. In the remainder of section 1, we introduce formally the model of 
signals used in the paper, and recall some basic results from random matrix theory, necessary for the next 
sections. In section 3, we introduce the subspace estimator of [16] and provide the main result of the paper, 
namely a CLT for this estimator, as well as numerical illustrations. Sections 4 and 5 contain the proofs of 
the results. 

1.2 Notations 

We introduce here the main notations used throughout the paper. 

The sets R, R“'" and N (resp. N*) will respectively represent the real numbers, the non-negative numbers 
and the non-negative integers (resp. the positive integers). C will be the set of complex numbers, and for 
z G C, Re(z), Im(z) and z* will stand for the real part, the imaginary part and the complex conjuguate. i 
will be the imaginary unit and we will also use the set = {z G C : Im(z) > 0}. The indicator of a set 
£ C IR is denoted 1^, d£ and Int(£l) will denote the boundary and interior of £. 

For a real-valued function ^p defined on R, supp((^) will represent the support of (p, and C“(1R,£) will the 
set of smooth compactly supported functions defined on IR, taking values in some set £ C R. 

Matrices (respectively vectors) are denoted by bolfaced capital (respectively boldfaced lower case) letters. 
For a complex matrix A, we denote by A'^,A* its transpose and its conjuguate transpose, and by tr (A) 
and ||A|| its trace and spectral norm. The identity matrix will be I. e„ will refer to a vector having all its 
components equal to 0 except the n-th equals to 1. 

^The material of this paper was party presented in the conference paper [12]. 
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The real normal distribution with mean m and variance is denoted A/'[r(q;, and the multivariate 
normal distribution in with mean m and covariance F is denoted in the same way A/K)!(m, F). We 
will say that a complex random variable Z = X + iY follow the distribution Afc{a + i/3, if X and Y 
are independent with respective distributions A/'[R(a, and A/k(/3, ^). The expectation and variance of a 
complex random variable Z will be denoted E[^] and V[Z]. The support of a probability measure /i will be 
denoted supp(/i). For a sequence of random variables and a random variable X, we write 

X and Xn X 

n—>-oo n—>-oo 

when Xn converges respectively with probability one and in distribution to X. Finally, = op(l) will stand 
for the convergence of to 0 in probability, and = Op(l) will stand for boundedness in probability 
(tightness). 

Some other special notations may be used at some very localized parts in the paper, and will be introduced 
in the text. 


2 Asymptotic behaviour of the sample eigenvalues 

In this section, we present some basic results from random matrix theory, describing the behaviour of 
the eigenvalues of the SCM , in the asymptotic regime where M, N converge to infinity such that 

^ c > 0. These results will be required to introduce the improved subspace estimator of [16]. To that 
end, we will work with the following random matrix model, refered to as ’’Information plus Noise” in the 
literature. We consider M, V, IF e N* such that K < M < N and M = M{N), K = K{N) are functions of 
N satisfying cat = ^ —>■ c € (0,1) as IV —)■ oo. For each N G N*, we consider the M x N random matrix 
Sjv, defined by 


Sat = Bat + W^v, 


( 2 ) 


with 

• Bat a rank K deterministic matrix satisfying sup at HBatH < oo, 

• Wat having i.i.d. entries Wij ^ Nc ^0, . 

We denote by Ai^tv > ... > \k,n > ^k+i.n = ■■■ = Xm,n = 0 the eigenvalues of BtvBJ^ (the non¬ 
zero eigenvalues are assumed to have multiplicity one for simplicity), and by Ui at, • ■., um,n the respective 
unit norm eigenvectors. Equivalently, > • ■ • > Xm,n are the eigenvalues of the matrix SatS^ and 
Ui^at, • ■ •, ujvA.Af the respective unit norm eigenvectors. 

2.1 The asymptotic spectral distribution 

Let /tAf = 'n!k=i „ the empirical spectral measure of the matrix SatS]^, with the Dirac measure at 
point X. From Dozier & Silverstein [5] [4], there exists a deterministic probability measure /i^v, with support 
supp(^Ar) C 1R“'', such that w.p.I., 

I^N — MiV -^ 0, 

N—¥oo 

where ”w” stands for the weak convergence. Equivalently the Stieltjes transform mN{z) of //at, defined by 

= L 

where Qn{z) = ~ satisfies for all z € C\[R 

mNiz) — rriNiz) > 0, 

N—¥00 
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where mN{z) = /j^ is the Stieltjes transform of which satisfies the equation 

mN{z) = ^tr Ta,(2;), (3) 

for all z G C\R, where the matrix TAr(z) is defined by 

TAr( 2 ) = { _ 2(1 + a'^CNmN{z))l + (J^(l - cat)!^ 

VI + CT"^CArTOAr(2) J 

Moreover, mN{z) can be further continuously extended to the real axis when z G —>■ x G R, and we 

denote the limit mAr(x). Defined in this way, x 1 —)■ mAr(x) is continuous on R, continuously differentiable on 
R\9supp(/rAr) and still satisfies the equation (3) for x G R\9supp(/XAr). 

We now recall the characterization of the support of /tat provided in [16]. Define the function wn{z) by 

Wn{z) = Z (1 + a'^CNmN{z))^ - CT^(1 - cjv) (l + Cr^CArTOAr(z)) , (4) 

The main equation (3) can be expressed in terms of an equation in wn{z), i.e. 

Z = (^Ar(w7v(z)), (5) 

where 

(l)Niw) = w{l - (T'^CNfN{w)f + cr^(l - CAr)(l “ a^CNfN{w)) (6) 

and 

/ivH = ^tr (B^B^-rcI)-'. (7) 

Starting from the properties that wn is real and increasing on R\supp(/iAr) and W]s[{x) G for x G 
supp(/iAr), [16] characterized wn^x) among the set of all solutions of the polynomial equation (j)N{w) = x 
(which has degree 2K + 2), for x G R, and showed that (j>N admits 2Q (1 < Q < 77 +1) positive local extrema 
^ 0 < xVat < N ^ N ^ N whose preimages are 

WNix^^N) < 0 < WNixtN) < ■ ■ ■ < WNiXq j^) < WNix'^^j^)- ( 8 ) 

Moreover, we always have W]\[{xq j^) > Xi^n, and if Q > 1, it turns out that for each q = 1,... ,Q — 1, there 
exists fc G {0,..., K} such that 

WN{Xg^j^),WNix~_^^ j^) G iXk,N,Xk+l,N) ■ 

By differentiating (5) on both sides, we find (j)'j^{wN{x)) > 0 for all x G R\supp(^Ar)- Finally, by showing 
that Im {wn{x)) = 0 for x G R\ U^=i[^^Af> I™ iwN{x)) > 0 for x G U^=i[2^^,Ar; [1®] concludes 

that the support of /x at is given by the union 

Q r _ ] 

Supp(/X7v) = IJ [Xg^N^^tN] ’ (9) 

9=1 

where the intervals x~j^,Xgjg are called’’clusters”. 

A typical illustration of function (() 7 v(u') for u; G R is given in figure 1. 

^Note that Q = Q{N) is a function of N 
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(p^iw) 



Figure 1: Typical example of the function <pN, for K = 3. For x G {^qN’^q+i n) with q = 1,. .. ,Q — 1, 
the equation (p^^w) = x admits 2K + 2 real solutions and WNix) is the unique solution in the in¬ 
terval (raAf(a;^^), jv))) for x < x^ (resp. x > Xg^), wm{x) is the unique solution in 

{—oo,WN{xPjq)) (resp. {wn{xq ^),oo)). Conversely, for x G supp(^Ar), the equation (p^iw) = x admits 2K 
real solutions plus two complex conjugate solutions, and wm{x) coincides with the solution having positive 
imaginary part. 


2.2 Useful quantities and bounds 

We now introduce a few bounds which will be of constant use for the derivation of the main results of the 
paper. Let us define 


fhN{z) = CNiriNiz) 



( 10 ) 


which corresponds to the Stieltjes transform of the probability measure CN^iN + (1 ~ cn)So- It can be shown 
that rfiN^z) = Tn{z), with 


=( 1 ■ ( 11 ) 

and note that w^iz) defined in (4) can be written as 

wn{z) = z (l -I- <j‘^CNmN{z)) (l -I- (j’^rfiNiz)) ■ 

The proof of the following bounds can be found in [9], [16] and [7]: matrices Tjv(z:) and Tjv(z) satisfy 


\T^n{z)\\ < 


C 


d(z, supp(^Ar)) 


and 


Tn{z) 


< 


C 


d(z, supp(/rAr) U {0})' 


( 12 ) 


where d{z,S) is the distance of z to a set S, and C,C are two positive constants independent of N,z. We 
also have, for all z G C, 


1-b cr^CArTOAr(z) <2, 


(13) 


and 


min |Afe,7v - W7v(-z)| > 


d(z,supp(/xjv)) 

2 


(14) 
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Note finally the two useful identities 


1 + <j‘^ciqmiq{z) = 


1 - (T‘^CNfN(wN{z)) 


and 1 + a‘^ci\fmi\f{z) = 


1 - a‘^CN fN{wN{z ))' 


(15) 


where Jm is defined by (7) and fN{w) = ^tr — wl) ^. 

To conclude this section, we introduce some quantities which will appear during the computations of the 
CLT. We define 


_ TAr(2i)B]vBJ^fTAr(2:2) 

N (I + a'^CNmiq{zi)) (I + a‘^CNmN{z2)) 


(16) 


as well as 

(^2 ^2 
vn{zi,Z 2 ) = Tm{zi)Tm{z 2 ) and vn{zi,Z 2 ) = 'iN{zi)'TM{z 2 ). 

Finally, we define 

A]v(zi,22) = (1 - UN{zi,Z2)f - Z 1 Z 2 V n{Zi, Z2)v N {zi, Z 2 ) ■ 

The last quantity lS.iq{zi,Z 2 ) satisfies moreover the following bounds. 

Lemma 1. For all Zi,Z 2 ^ supp(/i^) such that Zi yf Z 2 , Ai\[(zi,Z 2 ) can he represented as 


(17) 


(18) 


ANizi,Z2) = 


Zl - Z2 


Wn{zi) - Wn{Z2) ' 


(19) 


Moreover, if there exists a closed set £ independent of N s.t. supp(/rAr) C £ for all large N, and if 1C is a 
compact set s.t. 1C C C\ ({0} U £), then 


limsup sup \um{zi, Z 2 )\ < 1 

N^OO Zi,Z2G}C 


( 20 ) 


and 


0 < liminf inf |A^(zi, ^ 2 )! < limsup sup |A]v(2:i, ^ 2 )! < oo- 

Af->oo zi,Z2e/C AT-foo 


( 21 ) 


Finally, we also have 


An{zi,Z2) 

(1 - UN{zi,Z2)f 


< 1, 


for all Zi,Z 2 € 1C. 

Lemma 1 is proved in appendix 5.2. 


( 22 ) 


2.3 Separation of the sample eigenvalues 

In this section, we review some existing results concerning the location of the sample eigenvalues. 

The following terminology will be used in the remainder: an eigenvalue Xk,N of B^rB^ is associated with 
the interval of the support of /xat if WN{xfj.^) < Xk,N < 'a'Ar(a^^jv)- ^ turns out that the ’’noise 

eigenvalue” 0 is always associated with the first interval since < 0 < wn{x^j.^) (see 

(8)), which is thus called in this context ’’noise cluster”. Moreover, each ’’signal eigenvalue” Xi^n, ■ ■ ■, Xk,n 
is associated with a unique interval ^qN] for g = 1,..., Q ; in particular, a signal eigenvalue may be 

associated with the ’’noise cluster” while two signal eigenvalues may be associated with the same interval. 
We now introduce the two following additional assumptions, which informally ensure that the K signal 
eigenvalues Xi^n, ■ ■ ■, Xk,n will not be associated with the noise cluster, that is, will be separated from the 
’’noise eigenvalue” 0 (A;^+i, 7 v, ■ • • ,Xm,n) for large N. This assumption will be necessary to guaranty the 
consistency of the subspace estimator introduced in the forthcoming sections. 
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Assumption A-1: There exists 

0 < < liminfx^j"^ < limsupa;^^ < < liminfx^jY < limsupxi^ < t^- 

N-^oo ’ AT-J-OO ’ N-t-oo ’ AT-j-oo 

Assumption A-1 thus ensures that the noise cluster remains asymptotically separated from the the other 
intervals in the support of /xat, as A —>■ oo. From (8) and the fact that wn is increasing on R\/iAr, we have 
WN{ti) < 0 < WN{tf) < for all large N. The second assumption A-2 is related to the behaviour of 

the signal eigenvalues. 

Assumption A-2: For all large N, 0 is the unique eigenvalue o/BatB^ associated with the noise 
cluster, i.e. 

WN{t2 ) < Aat.at- 

Note that this assumption implies that lim inf at-j-oo ^k,n > 0, thus ensuring that noise and signal eigen¬ 
values are asymptotically separated (see lemma 2 below). These separation conditions have a direct conse¬ 
quence on the localization of the eigenvalues of the matrix Sat^^- Indeed, it was shown in [16] that under 
assumptions A-1 and A-2, 

^l,N, ■ ■ ■ ,^M-K,N & [ti ,t'^] and Xm-K+1,N, ■ ■ ■ ,^M,N & [t2 jtt]’ (23) 

with probability one, for N large, i.e. the ’’noise sample eigenvalues” split from the ’’signal sample eigen¬ 
values”. An illustration of the density of /xat and the localization of the sample eigenvalues (23) is given in 
figure 2. 



Figure 2: Density of fiN and sample eigenvalues of SatSat for one trial. The parameters are M = 10, 
N = 20, (7=1 and the eigenvalues of BArB[),f are 0 (with multiplicity 5), 5 (with multiplicity 2) and 10 (with 
multiplicity 3) 

Functions (pN represents in some sense a link between the support of fj,N and the eigenvalues of BatB^)^. 
Figure 3 shows the consequence of assumption A-2 on the behaviour of ^Ar(w) near w = 0. 


To conclude this part, we show that the separation conditions A-1 and A-2 ensure the effective separation 
between signal and noise eigenvalues of 

Lemma 2. Assume the separation conditions A-1 and A-2 hold. Then, 

liminf Xk.n > 0. 

N—*-oo 

Proof. Assume the converse. Then there exists a subsequence (p{N) such that Xk^^{n) —^at 0. From the 
condition A-2, we have for all large N 

WN{tf) < 0 < WN{tt) < WN{tf) < Xk,N, 
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Figure 3: Typical example of the behaviour of (f>N near 0, when assumption A-2 is satified (a), or not (b). 


and the condition A-1 ensures the existence oi x,y G ), with x < y, such that 

w^(N) {y) - (a;) —-^ 0. (24) 

iV—>-oo 

But using (19) and (21) in lemma 1 contradicts (24). □ 


2.4 The spiked model case: fixed rank 

When K is constant with respect to N, the results of the previous sections can be simplified. Indeed, in this 
case, we have for all 2 : e C\1R''", 


mN{z) — m{z) 


N —¥oo 


> 0 , 


(25) 


where m{z) satisfies the equation 


m{z) = 


—z (1 + a‘^cm{z)) + (7^(1 — c) ’ 


(26) 


and is the Stieltjes transform of the Marchenko-Pastur distribution [10], with support [(t^( 1 — Y^)^,cr^(l + 
An illustration of the Marchenko-Pastur distribution is given in figure 4. 


For any compact /C G C\ (, t)*"] U [^2 (25) can be strengthened with 

sup |mAr(z) — m( 2 ;)| -0, (27) 

ze/C Af-foo 

Simple algebra allows to rewrite the usual quantities in a simpler way. Indeed, we will have (in the same 
way as for (15)) 


1 -I- a‘^cm{z) 


w{z) 

w{z) -b cr^c’ 
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Figure 4: Density of the Marchenko-Pastur distribution in the case where c = 0.5, a = 1. 


with w{z) given by 

w{z) = z (l -b a‘^cm{z))'^ + a^{l — c) (l -b a‘^cm{z)) . 

(28) 

As for (5), equation (26) 

can be rewritten as 



(t){w{z)) = z, 

(29) 


where 


(w + a^c) (w + cr^) 

^ w) =- 

w 


(30) 


Of course, the boundary points of the support of the Marchenko-Pastur distribution, namely cr^(l — \/c)^ 
and cr^(l -I- Vc)^, coincides with the local extrema of (j), and with respective preimages 


w ((7^(1 — a/c)^) = and w (cr^(l -b -v/c)^) = 

The function w is continuous on R, real and increasing on R\[(t^(1 — -/c)^,(t^( 1 -b -s/c)^]- Figure (5) provides 
an illustration of the behaviour of the density of /xjv when K is fixed and N —>■ oo. In the case when K is 
fixed, the separation assumptions A-1 and A-2 have a useful consequence on the behaviour of the smallest 
non zero eigenvalue of 

Lemma 3. Assume K independent of N. Then, the separation condition A-1 and A-2 hold iff 


liminf Xk N > cP‘\fc. (31) 

N—¥00 ’ 


The proof of lemma 3 is defered in Appendix 5.3. 

In the special situation where the non-zero eigenvalues of B^rB^ converge to some different limits, i.e. 


Xk,N —Afe > 0 (32) 

for all k = 1,... ,K, with a^^/c < Xk < ... < Ai, it is shown in [9] that the number Q of clusters in the 
support of is exactly A -b 1 for A large and in this case, the ’’noise” eigenvalue 0 is the unique eigenvalue 
associated with the ’’noise” cluster Therefore, assumptions A-1 and A-2 are ensured in this 

case. It is also proved that ^ —>■ 4>{Xk), and using a refinement of (23), [9] also showed that (see also 
Benaych & Nadakuditi [2]) that the K largest sample eigenvalues Ai^tv, ■ • • , Xk,n split from the M — K 
smallest eigenvalues Xk+i,N) ■ ■ •) Xm,n and 


Afc, 


N 


N—¥00 

while N ^^(1 N ^^(1 — &.S. 


t <f{Xk), 


( 33 ) 
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(a) N = 20 


(b) N = 100 
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Figure 5: Density of when K = 2, cn = 0.5. The two non zero eigenvalues of are 5 and 10. 


Finally, we notice that A^v( 21 ,- 22 ) defined in (18) will satisfy, when K is constant 


sup |A7 v(2i,- 22 ) - A(zi, 2 : 2)1 -^ 0, 

zi,z2eK 


where A(2:i, Z 2 ) is given by 


A(2i,22) = 1 - 


ct'^c 


W{zi)w{z2)' 

The properties given in lemma 1 are of course valid for A( 2 i,Z 2 ): in particular, we have 

2l - 22 


A{zi,Z2) = 


10(21) - 1^(22) ’ 


for all 2i ^ ,22 and zi,Z2 ^ [o’^(l — v^c)^, ct ^(1 + as well as 

0 < inf \A{zi,Z2)\< sup |A(2 ;i, 22)! < 00 

Z\,Z2^K Z\,Z2^K 

and 

|A(2i,22) - 1| < 1, 

for all 2i,22 € K, with JC a compact set such that /C C C\ ({ 0 } U [cr^(l — + -x/c)^]). 


(34) 
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2.5 Contour integrals 

Thoughout the paper, we will deal with integrals of the form 

In = [ 'i/N{wN{z))w'j^{z)dz 

Stti JaK 


(35) 


where dlZ is the clockwise oriented boundary of a rectangle TZ intersecting the real axis at two points — e, 
+ e with e > 0 such that + e, and where 'I'jv is a meromorphic function with poles contained in 

the set {Ai^at, ..., Xk,n, 0}. It is shown in [16] from assumptions A-1 and A-2 that the set WN{dTZ) is a 
closed piecewise path intersecting the real axis at points — e), WN{t 2 + e), enclosing the non-zero 

eigenvalues of with winding number —1 and leaving 0 outside, for N large. Therefore, for all large 

A, a change of variable and residue theorem lead to 

1 f 

In = — f d^N^dw =-'^Res{d>N,Xk,N), (36) 


where Res('I'Ar,A) is the residue of d>Ar at A. Note that in the case of the spiked models (see section 2.4) 
where K is fixed with respect to A, and under assumptions A-1 and A-2, the previous result still holds by 
replacing wn{z) with w{z) and w'n{z) with w'(z). 


3 Noise subspace estimation 

In this section, we review the results of [16], [6] on the consistent subspace estimation, in the asymptotic 
regime where the number of antennas M = M(N) is a function of the number of samples A such that 
Cat = ^ c € (0,1) as A —)• oo. 


3.1 Consistent estimation 

Noise subspace estimation consists in our case in estimating the quantity 


K 


IJN = dlNl^Nd2,N = d^N d2,Ar, 


(37) 


k=l 


where (di^Af)j (d 2 ,Af) are two sequences of deterministic vectors such that supA? Ijdi^Arjl, supat l|d 2 ,Ar|| < oo. 


We recall that the traditional estimator based on the SCM ", " is defined by 


N 


Vn’ = dpAT I - X! W,ArU 


K 


k,N Q2,N- 


(38) 


k=l 


It was shown that under the separation assumptions A-1 and A-2, the quantity (37) can be written in terms 
of the following integral 


VN = dJ^Af ^ Tw(2): 


Wn^z) 


-dz d 


1 + a'^CN'ITlNiz) 
where dlZ is the clockwise oriented boundary of the rectangle 

Tl = {x + iy:x G [t^ - e,t^ +e], 2 / € [-<5,(5]} , 
with e > 0 s.t. + e < t 2 and 5 > 0. By defining 

wn{z) = z{l + CT^CArTOAr(z))^ — cr^(l -I- a‘^CNrhN{z)), 

it is shown in [16] that 




sup 

z^dn 


w'n{z) 


w'n{z) 


di,^QA.(z)d2.A.^^^^^-^^ - di,^TA,(z)d2.A.3^^5^-^^ 


N—¥(X) 


G 0 . 


(39) 

(40) 

(41) 

(42) 
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This of course readily implies that 


^ a.s. ^ 

Vn - Vn —-0, 

N—^oo 


(43) 


where 







1 + a'^ci^mi^{z) 


d0 d. 


i2,Ar- 


(44) 


Thus f}N is a consistent estimator of (37). 

Remark 1. The integrand in (44) is meromorphic with poles at Xi^n, • ■ •) Xm,n as well as at the zeros of 
the function z i—>■ 1 + denoted ... ,u!m,n■ It is shown in [16] that these zeros are the 

eigenvalues of the matrix ^ where Ajv = diag (^Xi^n, ■ ■ ■, Xm,n^ , and follow a property 

similar to (23), i.e 

■ ■ ■ ,d}M-K,N & [ti ,ti] and UJm-K+ 1,N, ■ ■ ■ ,djM-K,N & [t2 ^1^]^ ( 45 ) 


with probability one, for N large enough. This ensures that the integral can be solved using residue theorem, 
and an explicit formula in terms of Uk,N,Xk,N and uJk.N was provided in [16] for the improved subspace 
estimator (44). 

Remark 2. Originally, the estimator derived in [16] was based on the representation 

w'n{z) 


Vn = f d* ^TAr(z)d2,7V: 
Jdit 


idTZ 1 + a'^CNniNiz)^^’ 

where dlZ is the clockwise oriented boundary of the rectangle 

n = {x + iy :x e [tf - e,t[[ +e],y e [-<5,(5]} , 

enclosing the noise cluster (the contour (39) enclosing the signal cluster). In that case, (42) with TZ replaced 
by TZ still holds, and 


flN = f dj j,[QN{z)d2^N 
Jdn 


"»<*■) -d.. 


1 + a’^CNiJiM^z) 


Therefore, the subspace estimators of [16] and (44) coincide. We choose to keep the representation (44) 
(with contour enclosing the signal cluster), which will be simpler to analyze in the following. 

When K is constant, a simpler estimator of the localization function can be obtained [6] [16, Sec. C]. 
Indeed, since 


d„ = d:,„|i-T/ 


dn 1 + f^^cm(z) 


dz d 


l2,iV 


o(l), 


a straigthforward application of residue theorem leads 

Vn = rj^N +o(l) 

with probability one, where 


= d 


l,iV 


/ K \ 

I — ^ h d-k^NUk.N I d2,Ar, 

^ fe=i / 


(46) 


and where h{z) is given by 


h{z) = 


v'{z) 


1 + a‘^cm{z) 


w{z){w{z) + CT^c) 
w(z)^ — fJ^C 
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3.2 CLT 

3.2.1 The main result 

Before stating the main result, we need to introduce some new quantities. For k,i G {1, • ■ • ,M}, let 


'&N{k,e) = 

/ 1 ^ ^ 

Y I ^ 


(Zl, Z2)w'j^{Zi)w'j^iz2) 


on Ton i^k,N — Wn{zi)) {X^^n — Wn{zi)) {Xk,N — WNiz 2 )) {Xi^N — Wn{z 2 )) An(zi,Z 2 ) 


dzidz2, 

(47) 


with 


0n’^\zi,Z2) = 

ZlZ2 (l + a'^CNTTlNiZi)) (l + YcArmAr(2:2)) Y(zi,22) 

_l_ Xk^NXl^N^N^Zi, Z 2 ) 

(1 + a'^CNTUNizi)) (1 + a‘^CNmN{z2)) 

We define the 2x2 matrix ^N{kJ) by 

TNik,e) = 


+ {Xk^N + Xi^n) {1 — un{zi, Z 2 )) ■ (48) 


Re + i + „(bi)„(2.2) 

ylk,N 'k^N ) ^ 2 \'lk,N 'k^N ^ 'k,N 'lk,N 

-Tm 

yik,N 


—Im 


/ ( 1 , 2 ) ( 1 . 2 ) 
ylk,N 'k^N 


—Re 


Y(F2) (1.2)\ 1 / (1.1) (2.2) (1,1) (2.2) 

yik,N 'le,N J ^ 2 yik,N 'll,N + 'k,N 'lk,N 


where = d*^Ufc^Aru^ and we finally set 

M M 

r- = EE dN{k,(.)TN{k,l). 


(49) 


k=l 1 = 1 

The main result is the following. 

Theorem 1. Assume the separation conditions A-1 and A-2 hold. Then we have 


0 < liminf mindAr(fc,£) < limsup max'dAr(A:, ^) < 00 . (50) 

N—¥00 k,l N—¥00 

Moreover, if 0 < liminfAr ^ < limsupjv ^ < c, then 

liminf mini?Ar(A:,^) > 0, (51) 

N—¥(X) 


and if K is independent of N, then 


■dN{k,e) = 

a^CN {Xk.NXe^N + {Xk,N + A^_Ar)cr^ + Y) {Xk,NXe^N + ct^cat) 
2 (^^k,N ~ ~ iXk,NXe^N — CT^Cat) 


(1 — l[A:+i,M](^)l[lf+l.M](^)) + ^N{ky), 


(52) 


with maxfe_^ |eAr(A)^)| -^N 0. Finally, let {£,n) be a deterministic bounded sequence and denote = 
[Re (^at) , Im (Civ)]^- Then, 


Re (Cjv iVN - Vn)) = Ov iV ( Vy) ’ 

and if lim iniN > 0, it holds that 

Yv^%iEYBll-Y_^A/,(0,l). (54) 

The proof of theorem (1) is defered to section 4. 
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3.2.2 Discussions and numerical examples 


In this section, we discuss the consequences of theorem 1 and provide numerical examples illustrating the 
results. 

We first remark that in the statement of theorem (1), the purpose of the constraint liminfjv 
is to ensure that the fluctuations of fj^ — r]N are O Indeed, there exist several situations where 

the fluctuations can be faster than O For example, in the case where K is independent of N and 

= d 2 ,Ar = um.at, then we see from (52) that ef’rjvei = o(l) and thus 


Re {fjN - Vn) = Op 



The result of theorem (1) can be rephrased in a more precise way, by considering the fluctuations of the 
random vector [Ke {fjN — r]j^) ,1m {rjN — r]]\{)]^. However, in this case, we have to take into account the 
possible ’’degenerate” situations, when the covariance matrix r^r is asymptotically singular. Since it is 
difficult to state general results in this case, the next corollary focuses on one important special case where 
di^AT = d 2 ,Ar, i.e. the case of quadratic forms. 

Corollary 1 (Quadratic forms). Assume that di_jv = d 2 ,Ar = d^r, where (d^v) is a sequence of deterministic 
vectors such that limsup^ l|dAr|| < oo. Under the assumptions of theorem 1, 


VNif, N - Vn) 


V 




^)d)VU/c,ArU^_^dArd)(,U^_7vU| AfdAT 


N —yoo 


> AAp (0,1). 


The result of corollary 1 is illustrated in figure 6 by comparing the empirical distribution of the quadratic 
form Vn {riN' — Vn) (10® trials), where di^^r = d 2 ,Ar = gm, with the normal distribution A/p (0, 2i9Ar(M, M)). 
The parameters are M = 20, N = 40, tr = 1 and the matrix is diagonal with non-zero eigenvalues 

at 5 and 6. 



Figure 6: Empirical distribution of Vn {fj]\f — ry^v) (quadratic form) 


In the ’’non-degenerate” case, we have to ensure that F^v is asymptotically non-singular. By computing 
the smallest eigenvalue of Fjv, this is equivalent to 


lim inf 
N—^OC 


k,£. 


'VN{k,i)vtN^Vi]N 


k,e 


> 0 , 


(55) 


We therefore have the following result, by using the fact that 


Re {^NivN — Vn)) = [Re(CAf) 


Im(^jv)] 


Re(?7jv - Vn) 
-Im {fiN - Vn) 


(56) 
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Corollary 2 (Non-degenerate case). 


Under the assumptions of theorem 1 and if (55) holds, then 


Vn r-'/" 


Re {fjN - Vn) 
-Im(r)jv - w) 


V 


> A/iR2 (0,1). 


An illustration of corollary (2) is given in figure 7 where we have compared the empirical distribution of 
the bilinear form \/NRe {ffN — Vn) (10^ trials), where di_jv = gm and d 2 ,Ar = om-i, with A/r (O, ef r^vei). 
The parameters are M = 20, N = 40, u = 1 and the matrix is diagonal with non-zero eigenvalues 

at 5 and 6 {K = 2). 



Figure 7: Empirical distribution of VNRg {vn — vn) (bilinear form). 


3.2.3 CLT for the traditional noise subspace estimate 

To conclude section 3.2, we provide a CLT for the traditional noise subspace estimate, defined in (38) by 


At) 

V)v 


K 


= dt,Ar I - XI ^k,Nul,N dz.Af. 


k=l 


From (23), almost surely for N large enough, the K largest eigenvalues Xi^n, ■ ■ ■ /^k,n are located inside 
the rectangle TZ defined in (40), while the smallest M — K remain outside TZ. Therefore, for N large enough, 
almost surely, 


"nN = dpjv ^ Q]v(2)dzj da,AT. 


As for (42), we have 


sup |d); jv (Qjv(z) - TAf(z))d 2 ,jv| > 0. 


z^d'TZ 


N—^oo 


which immediately implies that 


where 


Define, as for (47), 

4^(fc,£) = 


'(t) (t) a.s. „ 

Vn - VW - -^ 0 . 


Vn’ = 7^ ^ di jvTAr(2)d2,Ardz. 

^’’’1 Jdn 


'^ 0 p^\zi, Z2) (1 + g-^CAfmAf(zi)) (l -f a‘^CNmN{z 2 )) _ 

27 riy Ion fan i^k,N - WNizi)) {X^^N - WNizi)) {Xk,N - Wn{z 2)) {Xg^N - Wn{z2)) IXn{zi,Z2) 


dzidz2, 


(57) 
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with A]^{zi^Z 2 ) and 0^j^'^\zi^ Z 2 ) defined respectively by (18) and (48). We define also 

M M 

=J2J2^N\k,i)TN{k,£), 

k=l i=l 

as for (49). Then we have the following result : 


(58) 


Theorem 2. Assume the separation conditions A-1 and A-2 hold, and that K is fixed with respeet to N. 
Then \nayik,i>K+i'd^N 0, 


<^(A:,£) = 


(Afe,jv + cr^) (a^_^ - a'^CN 

‘^^k,N (Afc.W + O'^Cjv) 


■ ^N{k,£) 


for k < K,£ > K + 1, with TLna'x.k<K,i>K+i \^Nik,£)\ -^n 0, and 

CT‘‘cArX^^(fc,£) 


£)^;^\k,£) = 


2Afc,ArA^,Ar {Xk,N + u'^cn)^ {^e,N + o-'^cn)'^ {£^k,N^e,N — ct'^cn) 


^(1) 


for 1 < k,£ < K, where x^N{k,£) is defined by 


XN^kJ) = 

Afc,ArA^,Ar [\k,N£^e,N + + ^i,N) + er'^) ((1 + C]\[){Xk ,NXe^N + ct^cat) + 2a^cN{Xk,N + Xt,N)) 

— c {Xk,NXe^N — O-'^Cn) {Xk,NXi^N + Cr‘^{Xk,N + Xi^n) + O-'^Cat) . 

Finally, let {£,n) be a deterministie bounded sequence and denote = [Re (^at) , Ini (^at)]^- Then, if 
liminfAT ^Afl^Af sAi ^ 0, it holds that 


y/N 


Re [fN (vn ~^n)) 


V 


pT p(*)e 

N SAT 


N —¥OQ 


t Vr (0,1). 


The proof of Theorem 2, which follows step by step the proof of Theorem 1, is omitted. 


4 Proof of theorem 1 

This section is dedicated to prove theorem 1. Several long computations will be defered to the appendix. 


4.1 Regularization and confinement of the eigenvalues 

To prove theorem 1, we will use the usual Levy’s theorem and prove the convergence of the characteristic 
function. Since the moments of fjN may not be defined, due to the poles in the integrand of (44) (see remark 
1), we first use a trick from [7], to force these poles to be away from the contour, and which does not modify 
the asymptotic distribution of fiN- 
Let ip € Cf° (R, [0,1]) s.t. 


ip{X) 


1 for A e [ti - + f] U [^2 - + |] 

0 for A e R\ ([tr - f + f ] U [^2 - + f ]) , 


(59) 


where e is given in (40), and dehne the regularization coefficient 

Xn = det if (SatS^) det tp (flN 


(60) 
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(see remark 4 for the definition of CIn)- From (23) and (45), we have xn = 1 w.p.l for N large and thus, 
for all p e N, we get 

Vnx'n = Vn + Op 



Therefore, to obtain a CLT for fy^vj we only need to prove a CLT for f]N^N- Moreover, it is proved in [7] 
that 


sup E 
zedn 


sup E 
zGdn 


dpTV iQN{z) — Tn{z)) d2,Ar 


w'n{z) _ w'ffjz) 

l + a'^CNmN{z) 1 + CT^CArTOAr(z) 



N ) ’ 

(61) 

— ) . 

(62) 


Since fluctuates less than the quadratic form dj f^QNiz)d 2 ^NXN, we can replace it with 

i+a^c^Jmiv(z) without modifying any asymptotic second order results. Indeed, it is easy to see from (62) that 


flNXN = dyjv ^ Qiv(^)XiV: 


w'j^{z) 


1 + a’^CNiriNiz) 

and the problem reduces finally to obtain the asymptotic distribution of 

w'n(.z) 


dz ) d2 V + Op ( — 


In = T-z <1 di jvQ]v(z:)d2,jvXAf:i—;—y 
27ri 1 + 


Remark 3. If VN (cn — c) 


sup 

z^TZ 


I an 

0, it can be proved that 

w'j^iz) w'{z) 


l + a'^CNTTiNiz) 1 + a^cm{z) 


CArTOAr(z) 


1 


dz. 


N 


(63) 


and the simplified estimator (46) derived in [6] will have the same asymptotic fluctuations as ripf. 
In the remainder, we denote by ipNiu) the characteristic function defined on R by 

ipNiu) = E exp ^m-\/]VRe (^AryAf)^ , 

where (^tv) is a deterministic sequence such that limsup^y |^7v| < oo. 

Finally, we recall two useful properties from [7, Prop. 3.3]: 

E [ 7 ^, - (dt^^d 2 .^ - w)] = O and E - {dl^d^^N - m)\" = ^ (^) ' 


(64) 


4.2 The differential equation 

We first prove that the characteristic function ipiqlu) satisfies the differential equation of a Gaussian char¬ 
acteristic function, up to an error term. 

Remark 4. Note that in the expression ofxpMiu), we ean assume for ease of reading and without loss of 
generality that f,N = I (by considering vectors di^jv^Ar and d 2 ,Ar^^ in the bilinear form). 

In the following, eAr(u, zi, Z 2 ) will denote a complex generic continuous function defined on R x dlZ x dTZ, 
such that u I—>■ eN{u, Zi, Z 2 ) is continuously differentiable, and 

limsup sup < |e7v(M, Zi, Z 2 )|, 

N^oo {zi,Z2)GdTZxdTZ I 

with P(w) a polynomial with positive coefficients. eAr(u,zi,Z 2 ) may take different values from one line to 
another. We will also keep the notation eAr(u), eAr(zi,Z 2 ), eN{u,zi) if eAr(t6,Zi,Z 2 ) does not depend on 
(zi,Z 2 ), u or 2 : 2 - 


d€N(u,Zi,Z2) 

du 


<p(u), 
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Using dominated convergence and Fubini’s theorem, the derivative writes 




{dlj,QN{z)d2,N + d;_^Qw(z)di,w) 


J 1 + 


so that we need to develop the term E [dj ■ By standard computations defered to 

appendix 5.1.1, we obtain 


V'7v(w) = 

I iv^Re (dj Ard 2 V - Vn) - i •r •r 9 

\ ’ 4 \2TnJ Ton fdTZ (1 + cr^CNmNizi)) {1 + a‘^CNmNiz2)) 

, <^n{u) 

Vn ’ 

where the quantity ( 21 , 22 ) is given by 

A<Ar(2l,22) = 


(/Xiv( 2 i, 22 ) +/iAf(^l,Z 2 ))lc)v( 2 l)w(v(^ 2 ) , j \ / N 

- jaziaz2 ipN[u) 


( 66 ) 


di,7vTAr(2i)Tiv(22)dj_lvE d*^QAr(zi)BArB^Qlv(22)dj^7VXAf. 


E 

*,1=1 
i¥^j 

+ d*^Tiv(zi)BArB^T7v(22)dj_ArE [d* jvQiv(zi)QAr(22)dj_lVXlv] 

+ SAr( 2 i, 22) (1 + a'^CNmN{zi)) (l + (j'^CNmN{z2)) d* jvTAr( 2 i)TAr( 2 : 2 )dj,JvE [d*^QAr(zi)Qjv(22)dj_ArXAr 

(67) 

and /iIV(- 21 , 22 ) is given by 
AAf(2l,22) = 


di,7vTAr(2i)Tiv(2;2)di,ArE d*_^QAr(zi)BArB^Qlv(22)dj^7VXAf. 


E 

*j=i L 

+ d*ivTAr(2i)BArB^Tjv(z2)di,ArE [d* jvQiv(2l)QAr(22)dj_lVXAr] 

+ SAr(2i, 22) (1 + (T^CivTOiv(2i)) (l + Cr^CivTOiv(22)) d* jvTAr(2i)TAr(2:2)di_ArE [d* ^Qiv(zi)Qiv(22)dj_ArXAr 

( 68 ) 


where sat ( 21 , 22 ) defined as 


B^Tiv(2:i)Biv B^Tjv(2:2)Bi\ 


/ \ _ l-\-a'^CNmN{zi) l+(7'^CNraN{z2) 

sn Zi,Z2 ^ (1 + a'^CNm.Nizi)) {1 + a‘^CNmN{z2)) 


(69) 


It now remains to approximate E [d*^QAr(2i)BArB)^Q7v(22)dj_AfX^] and E [d*^QAr(2i)Q7v(22)dj,AfXAf]: 
and we introduce for that purpose the following quantity 


rAr(2i,22) = q^tr 


TAr(2i)BArB^TAr(22)BArB 


N 


N ((1 + a‘^CNmN{zi)f ((1 + a'^CNmN{z2)f 
Recall moreover the definitions of un^zi, Z 2 ), vn{zi,Z 2 ) and ^^( 21 , 22 ) given by (16) and (17). 
Proposition 1. For all 21,22 G dTZ, we have SAr( 2 i, 22 ) +rAr( 2 i, 22 ) = 2 i 22 Wiv( 2 i, 22 ), 

E [d* jvQAf(2l)QAr(22)d2,ArXAf] = 

1 - MAr( 2 i, 22 ) .* rr. / irr. / IJ , Vn{zi,Z 2 ) d*_^T 7 v( 2 i)BArB^TAr( 22 )d 2 ,Ar eAf( 2 l, 22 ) 

A / \ AT -/V j -/V ('^2/^2 AT ' A / ^ I 2 /i i 2 / AT 

An{zi,Z2) An{Zi,Z2) (1 + a^CNmN{zi)) (1 + (J^CNmN{z2)) N 


(70) 


(71) 
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and 


E [d* jvQjv(zi)BArBJ^,QAr(22)d2,ArXAr] = 

(l + a‘^CNmN{zi)) (l + a'^CNmNizi)) {un{zi,Z2)sn{zi,Z2) + rjv(2:i, 22 )) 

/^n{zi,Z2) 


1 - Sn{zi,Z2)vn{Zi,Z2) - Un{Zi,Z 2) 
^Nizi,Z2) 


d* ArTAr(2:i)BjvB|^TAr(z2)d2,Ar ■ 


‘^l,AfTAr(^l)Tjv(z2)d2_Ar 

eN{zi,Z2) 

N 


(72) 


Note that the inverse of An(zi,Z 2 ) is well defined thanks to (21) in lemma 1. The proof of propo¬ 
sition 1 is given in appendix 5.1.2. Using the expression of E [d* ^Q 7 v(.Zi)B 7 vB^Q]v(- 22 )d 2 ,ArXAf] s-iid 
E [dJ_^QAr( 2 i)QAr(z 2 )d 2 ,ArXAr] in proposition 1 and the fact that sn{zi,Z 2 ) -f rN{zi,Z 2 ) = ziZ 2 Vn{zi, Z 2 ), 
we further obtain 


tJ'N{Zl,Z2) = 


E 

*j=i 


-f 


-f 


Z 1 Z 2 (1 + a'^CNmNizi)) (1 -I- a‘^CNmNiz 2 )) VAf ( 2 : 1 , 22 ) 

A7v(z1,^:2) 

VAr(zi, 22 ) (d*jvTAr(2:i)BArB^TAr(z2)dj,Ar) 

(1 -I- a’^CNniNizi)) (1 -I- Cr2cjvTOjv(^2)) AAr(zi, 22 ) 
2(l-MAr(2l,^2)) 


(d*,ArT7v(^:i)T7v(^:2)dj,Ar) 


A 


N 


j^TNiZi)TN{Z2)dj,Nd* j^TNizi)BNB%TN{z2)dj,N 

[Zl,Z2) 


eN{zi,Z2) 

N 


and 

P^n{Zi,Z2) = 


El 

*,i=i 


Zi 2:2 (1 + a'^CNmNizi)) (1 -I- (J^CArmjv(z 2 )) Vn{zi,Z 2 ) 

An{zi,Z2) 


di,7vTAr(2i)Tjv(22)di,Ard*^^Tjv(2l)T^(z2)dj_Ar 


VAr(zi, Z2)d*^T7v(zi)BArBJ^T7v(^2)di,Ard*_^T7v(^l)BArB^TAr(2;2)dj,Ar 

(1 -I- a‘^CNmN{zi)) (1 -I- 0-‘^CNmNiz2)) ANizi,Z2) 

2 (1 - UNizi,Z2)) 


An{zi,Z2) 


"‘^i.ArTAr(2:i)T7v(2:2)di,7vd*_^T7v(2:i)BjvB^Tjv(z2)dj,_/v 


eNizi,Z2) 

N 


Going back to (65) and introducing again the deterministic sequence {^n) (see remark 4), we finally obtain 


M M 


^iy/NRe {^N{dlffd2,N - Vn)) -u'^ ^dN{k,e) (^Re j ^n{u) 

(73) 


fc=i e=i 


V')v(w) = 

( 

, (^n{u) 

Vn ’ 

where we recall that = d* n^j.n and where dAr(fc,t') is given by 

’dNikJ) = 

— ( — V { i _ 9^^'^\zi,Z2)w'j^{zi)w'f^{z2) _ 

2 V27ri/ JdnJdn {^k,N - wn{zi)) {Xi^N - wn{zi)) {Xk,N - wn{z 2)) {Xi,N - wn{z2)) An{zi,Z2) ^ 

(74) 
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with 


Z 1 Z 2 (1 + (J^C7vmjv(^l)) (1 + Cr^CjvTO7v(^2)) Vn{Z 1 jZ 2 ) 

, )^k,N)^i,NVN{zi,Z2) , \ \ /I ! w 

+ Tf;- 2 - 1 --2- 1 -^ + M,n) (1 — Un{Zi,Z2)) ■ 

(1 + a^CNmN[zi)) (1 + a^CNmN[Z2)) 

We can check easily that '&N{k,£) G R. By letting = [Re(^Ar), Im(^jv )]^5 we obtain 

eNiu) 


where 


V’jv('w) = (^iv^Re (^jv(dt^jvd2,7v - Vn)) - u$,1fTN^N) '<Pn{u) + 


M M 


r- = EE 


k=l 1=1 


with the 2 x 2 matrix TM{k,t) given by 

TN{k,£) = 


Re 




- 2 )„( 1 . 2 ) 
N 'li,N 


( 2 . 2 ) ( 1 , 1 ) 

ylk,N 'li,N ' 'lk,N 'k,N 


—Im 


f„( 1 . 2 ) ( 1 . 

yik,N 'k,h 


—Im 


—Re 


f„( 1 . 2 ) ( 1 . 2 ) , 

yik,N ye,N j + 


(75) 


(76) 


f„(b 2 )„( 1 . 2 )' 

yik,N 'k,N 

\ {yk,'N 


.( 2 , 2 )J 1 . 1 ) 


(77) 


From the trivial inequality |Re( 2 ;iZ 2 )| < \ (kiP + \z 2 \'^) for ^ 1 , 2:2 € C, we have 


Re 


(f2 „( 1 . 2 )„( 1 . 

yiNVk,N 


< 




/ ( 1 . 1 ) ( 2 . 2 ) ( 2 . 2 ) ( 1 . 1 )\ 

yik,N 'k^N w 'k^N ) ’ 


which implies 

(h - -Ro .n(l’2)„(b2)A , \^n\" / (1,1) (2.2) (2.2) (1,1)\ „ 

N{k,£)^A! — He j+ ^ yik,N kk,N J — 

in other words that TA[{k,£) is non-negative definite. 


(78) 


4.3 Asymptotics of ^N{k,i) 

The purpose of this section is to prove (50), (51) and (52) for the coefficients 'dN{k,£). 

Using the bounds (12), (13) and the fact that \my{z)\ < d( 2 ;,supp(^]v))“^, it is easily shown that 
2 : 2 ), defined in (48), satisfies 


lim sup max sup 

N=f-oo zi,Z2GdlZ 


0^^’^\zi, Z2)wy{zi)wyiz2) 


< 00 . 


Moreover, from (14) and (21), we also have 


lim sup max sup 

N=^-oo z\,Z2^dTZ 


_ AAf(zi,22) ^ _ 

(Afc.AT — WAr(2l)) {Xe,N — Wn{Zi)) {Xk,N — Wm{z 2)) iXi,N — WNiZ2)) 


< 00 . 


Therefore, these bounds readily imply 


limsupmaxi?jv(fc,£) < 00 . (79) 

N=^oo 

We now express the integrand of 'dN{k,£) as a series of functions which are separable and symetric in zi, 22 
(i.e. a function 3 ( 21 , 22 ) is symetric separable if it can be written as 3 ( 21 , 22 ) = 3 ( 21 ) 3 ( 22 )). Notice that. 
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except for Aj^{zi, Z 2 ), all the functions appearing in the integrand in the definition of djv(fc, ?) are trivially 
sums of separable functions from their very definition. From lemma 1, we have 


AAf(zi,Z 2 ) 

(1 - UN{zi,Z2)f 


< 1 , 


and by writing 


An{zi,Z2) = 


(1 - UN{Zl,Z2)f ( 


1 - 


ZiZ2t>iv(zi,Z2)t'Jv(zpZ2' 

(l-Ujv(zi,Z 2 ))^ 


we obtain 


An{Zi,Z2) ^ 


E 


{ZiZ2Vn{zi, Z2)vn{zi, Z2))'' 

(1 - UN{zi,Z2) f^'''^ 


(80) 


Using this time the fact that \un{zi, ^ 2 )! < 1 (see lemma 1), we can further write 

An{zi,Z2)~^ ^ {ziZ2VN{zi,Z2)vN{zi,Z2)t UN{zi,Z2y^^-^^'^’‘"^^ ■ (81) 

fceN ii,...,i 2 fc+ 26 N 

Since the functions un, vn and vn are continuous on the compact set dTZ x dTZ, the bound previously 
derived shows that the series of functions defining (81) is uniformly convergent on dTZ x dTZ. Consequently, 
we can rewrite the coefficients dAr(fc,£), defined in (47), as 


'&N{k,£) = 

E E 

nGiNJ + 


1 f f a'^0^^’^\zi, Z 2 ) {ziZ 2 Vn{zi, Z 2 )vNizi, Z 2 ))'^ UNizi,Z 2 y^''~-~'~‘^"+^w'j^{zi)w'pf{Z 2 ) 
27ri/ Jdn Jdn 2 {Xk,N - wn{zi)) {Xe,N - wn{zi)) {Xk,N - wn{z 2)) (Xe^N - wn(,Z2)) 

(82) 


where {zi,Z 2 ) is defined in (48). In other words, we have written dN(k,£) as a convergent series of 
integrals of symetric separable functions. Consequently, 'diq{k,tj can be written as a series of squared single 

integrals, i.e. there exists a sequence of continuous functions (< 7 ^^) defined on dTZ such that 

V / 





2 


implying that ^^{k,^) > 0. This proves (50). 

To prove (51), we rely on the series expansion (82) introduced above. Using only one of the three terms 
in the definition of Z 2 ) (see (48)), and by only considering n = 0 in the sum of the series in (82), 

we obtain the following lower-bound 


'dN{k,t) > 

/ 1 V / / (T‘^Xk,NXl^NVN{zi,Z2) {1 +(T‘^CNmN{zi)) ^ (l -f Cr^CjvTOAr(^2)) ^'^w('2=l)^)v(-2^2) 

\27ri/ JdTZJdn {^k,N — wn{zi)) {Xe^^ — wm{zi)) {Xk^N — wn{z 2)) (Xe^M — wn{z2)) 

(83) 

From the definition of VNizi,Z 2 ) (see (17)), we have 


Vn{zi,Z2) 


M 


= -E 

M /A 


1 


(1 -I- a'^CNmN{zi)) (1 -f (j'^CNmN{z 2 )) N {Xm,N - Wn{zi)) {Xm,N - WNiz2)) '' 
and a usual change of variable gives 

n/f / \ 2 

dw \ 


4 ^ f- 


(dU) i^m,N — w) {Xk,N — w) {Xi^pf — w) 
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Residue’s theorem thus implies that for k,£ < K, 


'&N{k,£) > 


M - K 

N 2Afc,ArAf ,iv 


(84) 


In the same way, iov k^i> K 1, we obtain 
2 


f I f a‘^ZiZ2 (l + (j'^CNmNizi)) (l + a‘^CNmNiZ2)) VNizi,Z2)w'j^izi)w'^{Z2) ^ , 

VN{k,l) > f f ^-rvTT-- 7 -^- 7 -- 7 -r^d2;id2:2 

\2TnJ Jd'RJd'R 2 (Xk,N - Wn(Zi)) [Xl^N - Wn{Zi)) {Xk,N - Wn(Z 2)) [Xl^N - Wn(Z2)) 


^ 1 


E 


For k < K and i > K we have 

2 


(85) 


V27ri/ JdTzJdn ^Wn(Zi)Wn{Z 2) {Xk,N - Wn[Zi)) {Xk,N - Wn[Z2)) 


Using (80) and performing a serie expansion of (1 — un{zi, Z 2 )) we obtain as well 




cr"^ Xk,Nw'j^{zi)w'j^{z2) 


dzidz 2 = 


2Xk,N 


( 86 ) 


Tdn 2 wn{zi)wn{Z2) {Xk,N - Wn(,Zi)) (Xk,N - Wn{z2)) 

(51) will follow from (84), (85) and ( 86 ) and lemma 2 in section 2.4. 

4.4 Asymptotics of 

We now show (52), by assuming that K is independent of N. In this case, using the results of section 2.4, 
it is not difficult to show that 


max dNiky) — dKi{k,£) 


N—¥(X) 


0 , 


where 


^N{k,e) = ( dr 


27’‘iy fan fan (V.w - ^(^ 1 )) (A^^at - w(zi)) {Xk,N - w(z 2 )) (X^^n - wiz 2 )) 


(87) 


d2;idz2 (88) 


with m{z),w{z) and A{zi,Z 2 ) defined in (26), (28) and (34). From (81), it is clear that the following serie 
expansion also holds. 


A(2 i,Z2 ) ^ = 1 - 


4 

a^CN 


w{zi)w{z 2 ) 


= E 

new 


4 

O'^Cjv 

w{zi)w{z2) 


uniformly on dTZ x dTZ, and thus 

■dNiky) = 


nGN 




27ri 


(o-^ + ‘"EEEF + + ^b^v)) (g^Cjv)" w(zi) "^(2:2) '^w'{zi)w'{z2) 


I an Jan i^k.N - w{zi)) {Xi^n - w(zi)) {Xk,N - ^( 22 )) {Xe^N - w(z 2 )) 

By expressing the previous expression as square of single integrals, we obtain 
dN{k,£) = 


d 2 :id 2 ; 2 - 


— (ct^ + (Afc.AT + Xe^N)) F] (ct'^cat) 


neN 


H- ^Xk^nXe^N F] (o'^^cat) 


nexi 


^ / _ 1 _ 

27 J‘i Jw{dn) (V.AT - w) (Af^Tv - w) 

^ f _ 1 _ 

27 ri Jw(an) (•^fe.JV - w) {Xe^N - w) 


dw 


dw 
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Using classical residue computation, we eventually end up with (52), 

( t^cat {Xk,N^e,N + iXk,N + + a'^) {Xk,NXi^N + ct '^ ctv ) 


•dN{k,t) = 


2 (•^fe,Ar ~ 0 "^CAr^ {x^l,N ~ 0 "^CAr^ (Xk^NXl^N — CT'^Ctv) 


(l — l[x+l,M]{k)t[ii+l,M]{^)) ■ 


(89) 


We can easily show that 


liminf min 'd^{k,i) > 0. 

N-^oo 


and we also have the boundedness 


limsupmaxi?7v(fc)^) < oOj 

TV-s-oo 

which is ensured in the general case (K not necessarily hxed) by (79), but which also comes from lemma 3 
in section 2.4. 

4.5 Solution to the differential equation 

Recall that the differential equation (76) 


V'7v(^) = (^iv^Re {^N (di ^vd2.Ar - ijn)) - i>N{u) + (90) 

where {^n) is any deterministic sequence such that limsupjv |^Ar| < oo and = [Ite(^Ar),Im(^jv)]^ and 
u I—>■ eN{u) a generic continuously differentiable function such that 

limsup{|eAr(u)|, |e)v(u)|} < ?(«), 

N —¥oo 


with P is a polynomial independent of N with positive coefficient. By differentiating (90) with respect to u 
and using (64), one can check that 


E jv^Re {iN - (d);_Ard2,Ar -11n))) 
where jn is defined in (63). This implies 

R-e {^N {'In — (di_jvd2.Ar — w))) = 


— ^n'^n^n + ^ 


1 

y/N 


Op (^) if liminf AT > 0 

Op (^7^) otherwise 


which shows (53). By assuming that liminf at ^^Tat^tv > 0, we can replace ^n by 


modifying the boundedness property of eAr(u), and thus 

. VNRb {^n (dj Ard2,Ar — w)) 


V’(v(w) = i- 




- U IpNiu) + 


eAr(u) 

Vn ’ 


in (76) without 


(91) 


(91) being a classical nonhomogeneous linear differential equation of the first order, we easily obtain that 

= exp (-m^/2 ) + o(l). 


, VNRe {^N {'^N — {d-l ]s[^2,N — tin))) 
exp I m - 


^n^n^n 


which proves (54). 
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5 Appendix 


5.1 Proof of formula (66) and proposition 1 

In this section, we prove formula (66) and proposition 1 respectively. For that purpose, we use two tools, an 
integration by part formula and a Poincare inequality for Gaussian variables, which are well-known in the 
field of random matrix theory since the work of Pastur [13]. We first note that every function f : C C can 
be written as f{z) = / (Re(z), Im(z)). If / € C^(R^, C), we define the usual differential operators 

df{z) ^ 1 / df{x,y) .df{x,y)\ df{z) ^ 1 / df{x,y) .df{x,y)\ 

dz 2 y dx ' dy ) “ dz 2^ dx dy )' 


In this context, we say that / is continuously differentiable if / is continuously differentiable. The following 
lemma gives the integration by part formula and the Poincare inequality. 

Lemma 4. Let Zi = xi + iyi,..., Zn = Xn + '^y-a ben i.i.d. A/c(0,p^) variables and let f a continuously dif¬ 
ferentiable function defined on C" with polynomially bounded partial derivatives. Then, if z = {zi,..., ZnY", 
it holds that 


E [zkfi^)] 




dm 

dz^ 


and E [zfe/(z)] 




'dm' 

dzk 


Moreover, 


V [/(z)] 



df{z) 

2 

+ E 

5/(z) 

\ 

dzk 


dzk 



(92) 


Hereafter and in all the remainder of this appendix, eniu, zi, Z 2 ) will denote a generic continuous function 
on R X dTZ x dTZ such that u !->■ €n{u, zi, Z 2 ) is continuously differentiable and 

limsup sup < |e7v(M, Zi, Z 2 )|, 

A—j-oo {zi,z2)GdTZxdTZ I 


d€N(u,Zi,Z2) 


du 


<p(u), 


with P(u) a polynomial with positive coefhcients. ejv('u, zi, Z 2 ) may take different values from one line to 
another. We will also keep the notation e]s[{zi,Z 2 ) and e]s[{u,Zi) if eN{u, Zi, Z 2 ) does not depend on u or Z 2 . 
We recall the quantity jn, which is the regularized estimator defined in (63) by 


In — ^ j^Qi^{z)d2^NXNY 

Jdn 1 


w(v(z) 


-I- a^CNTHNiz) 


dz. 


Using lemma 4, it is not difficult to obtain, as in [7, Lem. 5.7], the following useful properties. 

Corollary 3. Let (/iAr)Ar>i be a sequence of continuously differentiable functions defined on mith 

polynomially bounded partial derivatives satisfying the condition 


limsup sup [hjv (Vec (QAr(z)), Vec(SAr)) xat] < 00 . 

N—¥(yo z^dTZ 


(93) 


Then, for all k G N*, we have 

E [hN (Vec (Qw(z)), Vec(S^)) 
=E [hN (Vec (Qw(z)), Vec(S^)) 
for all p G N. Moreover, 

W^^jhN (Vec (Qjv(^)), Vec(Sw)) 


eAf(u, z) 
NP 


N 


d 

dW,^, 


[hN (Vec (QAr(z)) , Vec(S^)) } xn 


(^n{u,z) 
NP ’ 


(94) 


(95) 
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and 


E \w,,jhN (Vec (Qw( 2)) , Vec(S^)) = 

^E[^{/i^(Vec(Q^(2)),Vec(S^))e‘“'^=^'^(E«)|^J (gg) 

L J 

Finally, we have 

E [hN (Vec(Qjv(^)) ,Vec(Sw)) 

and 

E[/i^(Vec(Q^(z)),Vec(S^))7^*/“'^^'^(E«)] = 

for all p e hi, with Dij = {x^}- 

We now introduce the matrix Rjv(z) given by 

[m„(«)x»l) + ..^(1 - «)) . 

Matrix R 7 v( 2 ) is similar to T 7 v(^) where we just replaced iriNiz) by E[TOAr( 2 :)xAr]- Since E[mAr( 2 )xAf] ~ 
nii^iz) —tjv 0, it is of course expected that Rn{z) will be close to T7v(^) asymptotically. This result is given 
by the following lemma. 

Corollary 4 ([7, Lem. 3.10, 5.5 & 5.6]). Let MAr( 2 ) a sequence of deterministic matrices of size M x M 
such that 


limsup sup ||Mjv( 2 :)|| < oo. 

N—^oo z^dlZ 


Then we have 


limsup sup |l + (T^CArE[TOAr(0)xAf]| < oo and limsup sup || Rat( 2 )|| < oo. 

N—>oo zGTZ N—>-oo zGTZ 

Moreover, E[TO]v(z)xAf] — '>tin(z) = and 

dt^^(E[Qjv(z)xw] - IlN{z))MN{z)d 2 ,N = and dj_^(R^(z) - Tjv(z))MAr( 2 )d 2 ,iv = 

We now give a result on the variance of certain expressions involving the resolvent, whose proof is a 
standard application of the Poincare inequality (see e.g. [7, Lem. 5.8], [16, Lem. 10]), and is therefore 
omitted . 

Corollary 5. Let MAr( 2 i,^ 2 ) a sequence of deterministic matrices of size M x M such that 

limsup sup j|M]v(zi, 22)11 < 00 . 

iv-xoo (zi,z2)edn^ 

Let P a 5-variate polynomial function independent of N such that the matrix 

E.m{Zi,Z2) = P (Q7v(zi), Q7v(^ 2), S^, M]v( 21 , Z 2 )) XN■ 

is properly defined. Then, it holds that 


limsup sup V [tr 2^2)] < 00 and limsup sup V -v/iVd* -22)d2^7V < oo. 

N^oo {zi,Z 2 )edn^ Af-xoo (zi,z2)ea7?,2 L ' ’ J 

Moreover, it also holds that V dj jyH]v( 2 i, 22)d2,Are'“'^^®^Eiv) ^ term behaving as eAr(u, 21,22). 
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5.1.1 Proof of formula (66) 

By expressing the derivative of w.r.t. Wij, we obtain 

^ ( AuVNRe(jfi)\ _ 

r / 

{-dlj,QN{z)e,e*-E%QN{z)d2,NXN + 

zzmjQT^^ ■< l + a^CNmN[z) 

w'n{^) 3i«yiVRe(7„)^^ 

(97) 


it6 1 


+ ~2'^[J {~^2,NQNiz)eie*'E*f^ClN{z)di^]S[XN + d2_^QAr(^)di_Aril*^) 2 _|_ ' 


where Dij is defined in corollary 3. The derivative with respect to Wi^ is computed in the same way. To 
develop E d);_jYQ]v(z)d 2 ,ArXAre‘“'^^®^^"^ , we start with the classical resolvent identity Q]v( 0 )SArS^ = 
I + zCIn(,z). By applying corollaries 3 and 4 several times, long but straightforward computations lead to 


a%'^\u,zi) + 


ma 


2 1 f (^P‘}!j’^\u,Zi,Z2) + P‘'n^\u,Zi,Z2)^ W'^(Z2) 


- [An{u,Zi\^ + 


2\/N 27ri Jqt^ (1 + ct^catE [m7v(2:i)XAf]) (1 + cr'^CNmN{z 2 )) 

tN{u,Zi) 


d2:2 


NP 


for all p € N, where Zi) is given by 


aN^\u,zi) = 






1 + ct^catE [m7v(zi)XAf] 


E 




(1 + ct^catE [m7v(2:i)XAf]) ^ 


1 + Cr2cArE[TOAr(2:i)X7v] 


tr BJ^fE [QAr(2i)xAf] Bat, 


zi, Z 2 ) and {u, zi, Z 2 ) respectively by 
Pn^Hu, Zi,Z2) = 


5(bi 


2 AuVNRe{^N) 


di,ivQAr(^2)eje*Q7v(^i)BjvS^Qjv(z2)d2,ivXA^6 

d2,ivQiv(^2)e,e*Q^(zi)BArS;.Qw(z2)di,ArXwe™'^^'^^^”^ 


(98) 


and 


Pn^Hu,Zi,Z2) = 


dlj^qN{z2)'^N'S%eje*QN{zi)ClNiz2)d2,NX%e^"^^^^^-^ 


+ E 


dl^Qw(22)SwS^e,e*QAr(0i)Qjv(^2)di.wX?^e'“'^^®(^~) 
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and where finally the matrix A]v('u, 2i) is given by 
ANiu,Zi) = 


E 




(1 + ct^catE [mNizi)xN]) 
1 + ct^catE [mAr(2i)Xiv] ^ 
1 + ct^catE [mAr(2i)Xiv] ^ 




/ ^2 

i ^tr QAr(2i)xiv - E 


—tr QAr(2i)xAf 


—tr S;^Q7v(^l)BArXJV 


— tr S^QAr(2l)BArXAf - E 


—tr i:*f^ClNizi)BNXN 


—tr CIn{zi)xn - E 


— tr QAr(2l)XAf 


(99) 


From (98) and the resolvent identity, we obtain 

BatB^ 


E 




1 + CT^CatE [TOjv(2:i)XAf] 


- Z (l + cr^CArfAr(zi)) I 


[I]*,jXwe 


in\/iVRe(7iv) 
"t" [^iv(^5 ^l)]ij + 


iiicr^ 1 


(*j)/ 

N 


{u,Zi,Z2) + PN^\u,Zi,Z2)]w'pf{z2) 


2y/N 27ri Jq^ (1 + ct^catE [rTiAr(zi)xAf]) (1 + (r‘^CN'mN{z2)) 

<^n{u,Zi) 

Np ’ 


dZ2 


where 


TAr(-2l) =- 


1 - ^tr 


B^E[Q^(zi)]B^ \ 

z(l + Cr2cArE[TOAr(Zl)XJv]) V" Vl + O-^CjvE [mAr(Zl)XAr]/ 

Straightforward algebra gives 


fN{zi) = E 
and finally we get 


mAr(zi) - 


(1 — I/cat) 


Zl 


XN 




E 


dlj^QN{zi)d2,NXNe'^^^<^-^ 


d* ArBAf(^l)d2,ArE 


Xnb 


mx/7VRe(7iv) 


iua^ 1 


(^Pn{u ,Zi,Z2) +/3jv(m, -^1,22)) w)v(^2) 


2y/N 27ri (1 + cr^cj^E [mAr(zi)xAr]) (1 + a‘^CNmN{z2)) 

^ , ^n{u,Zi) 

\ ^n{0^z,)+ . 


dZ2 


+ dlj^A^iu, zi)R^(z2)d2.jv + E [dJ^jvQiv(2i)d2.7ve™'^^"(^«^ 

where (3n{u, Zi, Z2) is defined by 
I3n{u,zi,Z2) = 


( 100 ) 


E 


d* ArQw(^2)R-v(^i)d2,AfdJ_^QAr(zi)BArS^QAr(z2)d2,ArXAre 


,2 iuy]VRe( 7 jv) 


dJ_ArQw(^2)SArS)(rR7v(^l)d2,Ardi AfQAf(^l)Qw(^2)d2,ArXAf6' 


2 i«vdVRe(7N) 


and Pn{u,zi,Z2) by 
Pn{u,Zi,Z2) = 


E 


dl^Qw(z2)Rv(zi)d2,wdt,^QAr(zi)B^S^QAr(z2)di.^X>‘“'^^'^^^”^ 


d2_7vQAf(^2)SArSArRAr(^l)d2,Ardi ArQiv(^l)QAf(^2)di,ArX7v6‘ 


2 i«vdVRe(7N) 
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Using corollaries 4 and 5 (in conjonction with Cauchy-Schwarz inequality), and the fact that = 

iPn{u) + O {N~P) (by dominated convergence theorem, see section 4.1), it is straightforward to show that 


* ^ 1 /* 


(^Pn{u, Zl,Z2) + PnIu, 21 , 22 )) w'f^{z2) 


dTZ (1 + Cr2cArTOjv(2l)) (1 + CT2cArmAr(22)) 


dZ2 


ejv(M, 2i) 

N 

( 101 ) 


The next step consists in decorrelating the different terms inside the expressions of (3n and /3n- We have 


E[dJ^^Qiv(22)Rv(2i)d2,ivdLivQw(2i)BvS^Qiv(22)d2,ivXwe‘“'^^'^^^”^ 

E [dJ_^Qlv(22)Rjv(2l)d2,ArX./v] E [dJjYQAf(2l)BvS^Qlv(22)d2,ArXAf] i’Niu) 

+ E [dlj, (Qiv( 22) - E[(Qiv(22)])Riv(2i)d2.ivd*_jvQiv(2i)BivS^Qiv(22)d2,ivX^e™'^=^'=(^«)' 
+ E [dJ_^Qv(22)Riv(2i)d2,ivXw] E [dlj, (Qiv(2i)BArS^Qiv(22) - E [Qiv(2i)BArS;.Qiv(22)j) 
and using again corollaries 4 and 5, we end up with 


2 i«U]VRe( 7 jv) 


di_jvQiv(^2)RAf(2l)d2,ArdJ jYQiv(^l)RAf^wQAf(^2)d2,ArXAre 

E [d* jYQiv(22)RAr(2i)d2,ArXAf] E [dj jvQv(2i)BivS^Qlv(22)d2,ArXAr] 


ejv(M, 21,22) 

Vn 


In the same way. 


E 


dLivQw(22)SivSwRiv(2i)d2,ivdLivQiv(u)Qiv(22)d2.ivX?^e‘“'^^‘^(^«)] = 

E [di_ivQw(^2)SArS^Rjv(zi)d2,ArXAr] E [d*jYQAf(2l)Qlv(22)d2,ArXAf] 4’n{u) ■ 
A standard application of corollaries 3 4 and 5 leads to 


ejv(M, 21,22) 
VfV 


E [d*jvQw(^l)Bv5]^QAr(z2)d2,ArXAf] = 

E [d);^^QAr(zi)BivB^Qlv( 22 )d 2 ,ArXAr] E [dJ_^Qlv( 2 i)Qlv( 22 )d 2 ,Ar] ^tr B^E [Qv( 2 i)x 7 v] Bjv 

1 + CT^CatE [TOAr(22)] (1 + CT^CatE [TOAr(2i)]) (1 + (T^cjvE [mAr(22)]) 

eAr('u, 21,22) 

N 

and 


E [d*jYQiv(22)SAfS^RAr(2i)d2,AfXw] = 
E [di^AfQlv(22)BArBJ^RAr(2i)d2,Ar] 

1 + ct^catE [mjv(22)] 

/^tr B^E [QAr(22)] Bat , cr 


(1 + a^CArE [mN{z2)]) ^ + ct^catE [mN{z2)] 


E[d^ ArQ7v(22)RAr(2i)d2,iv] 


eAf(u, 21,22) 

7 V 3/2 


( 102 ) 


Inserting the previous estimates into the expressions of I3m(u, 2 i, 22 ) and I3m(u, Z\, 22 ), and replacing E[miv( 2 )] 
by mN{z) as well as E[QAr( 2 )] and Riv( 2 ) by TAr( 2 ) thanks to corollary 4, we finally obtain (66). 


5.1.2 Proof of proposition 1 

Since the proof of proposition 1 uses the same technic as in the proof of formula (66) (see appendix 5.1.1), 
we will only provide the main lines of the computations. 
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Let M7v(2i, ^2) be a M X M deterministic matrix s.t. 


limsup sup ||M7 v(2:i, 2 ^ 2)11 < oo- 

N^oo (zi,Z2)GdTZxdTZ 


We will also use the generic notation 'E,m{zi, Z 2 ) for M x M matrices such that 

limsup sup |tr EAr(zi,Z 2 )| < 00 , 

N—^oo (zi,Z2)GdTZxdTZ 


i.e. such that tr E 7 v(- 2 i,^ 2 ) behaves as e]\[{zi,Z 2 )- The value of Ejv(^i, 2 : 2 ) may change from one line to 
another. 

Starting from the matrix E [QAr(zi)MAr(zi, Z 2 )QAf (^ 2 )XAf], a repeated use of corollaries 3 4 and 5 together 
with the decorrelation trick, in the same way as in appendix 5.1.1, leads to 


Z 2 (1 + CT^CatE [to]v( 22 )]) I - - cr^ I- 


Qjv(zi)MAf(zi, 2 : 2 )Qjv(- 2 : 2 ) 

Z2 (1 + CT^CtvE [mAr(z2)]) 

- -E [[Qa,(zi)Ma,(^i)] Xn] - ^ F 

22 22 (1 + Cr^CArE[mAr(22)J) 


N ' 


tr B;^E [CIn{z 2 )] Bat 


1 + ct^catE [m7v(^2)] 


XN 


N 


tr QAr(- 2 l)MAr( 2 i, 22 )QAr(- 22 )XAf 


E [Qv(2i)Xiv] 


ra2 


22 (1 + ct^catE [mAr( 2 i)]) (1 + ct^c^E [m7v(^2)]) 
E [QAr(2i)xAf] ^tr B;^E [QAr(^2)] B^v 


+ 


22 (1 + ct^catE [TOAr( 2 i)]) (1 + ct^catE [mjv(^ 2 )j) 
Eat(^1, 2 : 2 ) 


— tr QAr( 2 i)MAr( 2 i, 22 )Qv( 22 )BjvB;^XAr 


— tr Q7 v(-2i)M7v(2i,22)Q7v(-22)XW 


+ 


iV2 


By introducing the matrix Rat( 22 ) and as in appendix 5.1.1, we obtain 


E [QAr(2i)MAr(2i,22)Qv(22)] = 

E [[QAf(^l)MAr( 2 i)RAr( 22 )XAf] Xiv] 


E [Qv(2^l)EArSj^RAr(22)XAf] 


1 + cr^cjyE [mAr(22)] 


N 


tr QAr( 2 i)MAr( 2 i, 22 )QAr( 22 )xJV 


_ E[Qjv( 2 i)xAf] _ 

(1 + Cr2cArE[mAr(2i)]) (1 + CT^CatE [mjv(2:2)]) L-^ 

_ E [QAf(zi)Xjv] [QAf(^2)] Bat 

(1 + cr^cjvE [mAr(2i)]) (1 + ct^catE [rhN{z2)] f 
EAr(2i,22) 

+ iV2 


tr QAr(2i)MAr(2i,22)QAr(22)BArB^X7V 


— tr Qv( 2 :i)MAr( 2 i, 22 )Qjv( 2 : 2 )XAf 


By taking the trace in the previous expression, and using corollaries 3 4 and 5, we end up with the following 
2x2 linear system 


^tr Q7v(2:i)Mjv(2:i,22)QAr(2:2)xw 


E 



Qw (^i)Mjv(2:i,2:2)QAr(^2)BivB^ 

{l-\-(T^CNmN{zi)){l-\-<y^CNmN{z2)) 


VN{zi,Z 2 )sNiZi,Z 2 ) +UNizi,Z 2 ) 

VNiZi,Z 2 ) 


E 

^tr QAr(2:i)MAr(2i,22)QAr(2:2)XW 

1 

un{zi, Z 2 )sn{zi, Z2) + r^izi, Z2) 

Un{Zi,Z 2 )_ 


E 

<T^ i Qjv(zi)Miv(zi,Z 2 )Qjv(z 2 )BivB 3 ;, - 1-1 

N ^ (l+(T 2 cjvmjv(zi))(l+(T 2 cjvmjv(z 2 )) 


^tr Tat ( 21 ) Mat ( 21 , 22 ) Tat ( 22 ) 

1 

eNizi,Z2) 

^tr TAr(2i)M7v(2i, 22)TAr(22)BArBJ^, 

+ XP 

_ejv(^l, 22 )_ 


where UAr( 2 i, 22 ), UAr( 2 i, 22 ), rAr(2i, 22 ) and sn(zi,Z 2 ) are respectively defined in (16), (17), (70) and (69). 
The determinant of the previous system is given by 

AAr(2i,22) = (1 - UN{Zl,Z2)f - Vn{zi,Z2) {sn{zi,Z2) + rAr(2i,22)) . 
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By relating Tjv(2:) with Tjv(z), we obtain the equality 

BjYTjv( 2 )BAr /i I 2 i' ffi i' ^ 

——5-- =1 + zll + a CNmN { z )) Tn { z ). 

1 + a'^CM'rriMyz) 

Inserting relation (104) in the expressions of rAr(zi,Z 2 ) and sn{zitZ 2 ), we obtain respectively 


(104) 


Sn{zi,Z2) = 


G 


TAr(2:i) 


G 

- Z2^tr 


T^n{z2) 


(1 + (j'^CN'mNizi)) {I + (j‘^CN'mN{z2)) N I-\-(T'^CNmN(,Z2) N I + a’^CNiriNizi) 

and 

rN{zi,Z2) = 

ZiZ2Vn{zi,Z2) - 


—a 


a 

+ -^ii^tr 


TAr(zi) 


a 

+ 2:2 ^tr 


TAr(z2) 


{I + a‘^CNmN{zi)) a‘^CNmN{z2)) ' N 1 + CT^CArTOAr(z2) N 1 + a‘^CMmM{zi)' 
where vn{zi,Z 2 ) is defined by (17). The determinant thus writes 

An{zi,Z2) = (1 - Ujv(2:i,-2:2))^ - ZiZ2Vn{Zi, Z2)vn{zi, Z2). 

Using lemma 1, we can finally solve the system (103) to obtain 


E 


— tr QAr( 2 l)MAr(zi,Z 2 )QAr( 22 )XAf 


(1 - UNizuZ2)) ^tr Tjv(zi)Mjv(zi, 2:2)Tjv(^2) + VNizi,Z2)'^tT , eNiZi,Z2) 


ANizi,Z2) 


+ 




and 


QN{zi)MN{zi,Z2)ClN{z2)BNB*ff 


:XN 


N {1 + a^CNTTiN^Zi)) {1 + a^CNmNiz2))' 

{uNiZi,Z2)sN{zi,Z2) + (^i, Z 2 )) fytr TAr(2i)MAr(2:i,Z2)T7v(z2) 


AAr(zi,Z2) 


(1 -gtr , «(«.«) 


+ 


ANiZi,Z2) 


7V2 


The approximations (71) and (72) will unfold by choosing 'M.iq(zi,Z 2 ) = di,Ard2jy, whcih concludes the 
proof of proposition 1. 


5.2 Proof of lemma 1 

We recall here that Am{zitZ 2 ) is defined (18) by 

AAr(2;i,Z2) = (1 - UN{z-i,Z2)f - Z- 1 Z 2 V n{Zi, Z2)v N [zi, Z 2 ), 

with un{zi,Z 2 ), vn{zi,Z 2 ) and vn{zi,Z 2 ) given by (16) and (17). We also recall the following bounds from 

[ 6 ] : 

sup sup |uAr( 2 :, z*)| < 1 and inf inf |AAr(2,2:*)| > 0. (105) 

N^oozeK N^oozeK 

For Zi,Z 2 G die, Cauchy-Schwarz inequality and (105) gives 

\uNizi,Z2)\ < \uNiZi, zDl^^"^ \uNiz2, Z2)\^^'^ , (106) 

and thus 


limsup sup |uAr(2:i, Z 2 )| < 1, 

N^OO Zi,Z2^K 
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which shows (20). In the same way, since 


\An{zi,Z 2 )\ > (1 - |ujv(zi, 22 )|)^ - \zi\\z 2 \ |uAr (^1 , ^ 2 ) | |iijV (^l, ^ 2 ) | , 
a straightforward application of Cauchy-Schwarz inequality yields 
|AAr(zi,Z2)| > 

(l - \un{Zi, zl)\^^‘^ \uNiz2,Z2)\^^'^'^ - \zi\\z2\ (zi, Z* ) | |wAr (Z2 , Z 2 ) (zi, \v N {zi, zl)\^ ^ . 

Using the inequality 


(1 - ^ 3 ^ 1 X 2 )^ > ^(1 - Xif - (1 - X2f - S2t2, 

valid for Xi G [0,1] and (1 — Xi)^ > Siti (i = 1, 2), we finally obtain 

|AAr(zi,Z2)| > |AAr(zi,Z*)|^/^ | A^v (z2, Z 2 ) | , 

which readily implies 

inf inf |A 7 v( 2 :i, Z 2 )| > 0. 

N^oo zi ,Z2G}C 

Moreover, from (12), and the definition of un{zi, Z 2 ), vn{zi,Z 2 ) and Z 2 ), we also see that 


(107) 


|A]v(zi,Z 2 )| < P 


1 


Q 


1 


^d(zi,supp(^]v) U {0})/ Vd(z 2 ,supp(/rAr) U {0})/ ’ 
where P, Q are two polynomials independent of N, zi, Z 2 with positive coefficients, and we thus deduce 

sup sup |AAr(zi,Z 2 )| < 00 , 

N^OO Zi,Z2GK 


(108) 


which shows (21). We now prove (19). By straightforward computations, it is easily shown that A]v(zi,Z 2 ) 
is the determinant of the following 2x2 linear system 


aCN (to7v(^i) - TOAr(z2)) 


Un{zi,Z2) 

Vn{Zi,Z2) 

acN (mAr(zi) - mN{z2)) 

2l - Z2 

Vn{Zi,Z2) 

a (ziTOAr(zi) - Z2m7v(-Z2))_ 


_ZiZ2U7v(zi,Z2) 

Un{zi,Z2)_ 

a (zimAr(zi) - Z2mAr(z2))_ 

“T 

a 

_MAr(zi,Z2)_ 


Since AAr(zi, Z 2 ) 7 ^ 0 for Zi, Z 2 € 1C, solving the previous linear system gives 


acN {mN{zi) - mAr(zi)) 


Zl - Z 2 Ujv(zi,Z 2 ) 
AAr(zi,Z2) <T 


and it is easy to show that 


aCM (TOAr(zi) - mAr(zi)) 


! t \ ( -.^Vn{zi,Z2) 

{Wn[Zi) - Wn(Z2)) - 


Thus we obtain the relation 


Wn{zi) - Wn{z 2 ) = 


Zl - Z2 


AAr(zi,Z2) 

which shows (19). Finally, to prove (22), we write, using Cauchy-Schwarz inequality, 

AAr(Zi,Z 2 ) - (1 - WAr(Zi,Z 2 ))^ = |zi| |Z 2 | |uAr (zi, Z 2 )wjV (zi, Z 2 ) | 

< |U| \z2\vN{Zl,zlY/'^VM{z2,Z2f/‘^VM{Zl,zlY/'^VN{Z2,zlY/'^, 
and since (1 — un{z, z*))'^ > \z\'^vn{z, z*)vn{z, z*) for z € /C, we have 


AAr(Zi,Z2) - (1 - ■UAr(zi,Z2))^ < (1 - Un{zi,zI)) (1 - U7v(z2,Z2)) . 
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As |1 —MAr(2i, ^2)1 > 1 —|■w^r(zl, z*)|^/^|MAr(z 2 , 9'^'^ thanks to the inequality a/( 1 — a)(l — 6) < l — \/^ 

valid for a,b G [0,1], we finally obtain 


or equivalently 


An{Zi,Z2) - (1 - MAr(2l,Z2))^ < |1 “ MW (-Zl, ^ 2 ) | ^ ■ 


An{zi,Z2) 

(1 - UN{zi,Z2)f 


< 1 . 


5.3 Proof of lemma 3 

Assume that the separation condition A-1 and A-2 hold, and let t G Since ijn converges to the 

Marchenko-Pastur distribution, rcAr(t) > 0 —)■ w{t) > 0 and we deduce that t > (1 + Vc)^. From (29) and 

the behaviour of </>, we have w{t) > a'^^fc, and finally assumption A-2 implies that liminf^v > w{t), 

which proves (31). 

Now, assume that (31) holds and let e > 0 such that 

liminf Ai^ TV > + e. (109) 

N—¥CX) ’ 


For any compact JC C (— 00 ,0) U (O, + e), we have 


sup 

w^K 


fN{w) 


1 

w 


-^0, 

N—^oo 


sup |^;'Tv('u;) - ^(w)! —-^ 0, 

weK N-^oo 


and sup \4>'j^{w) — 4>'{w)\ -)• 0 (110) 

weK N-^oo 


Since ^(w) has a unique maximum cr^(l — ^/c)^ at point w = —a^sfc on the interval (— 00 , 0), will also 
admit a positive maximum in this interval for all large A, and thus 

= (7^(1 - v/c)^ + 0 ( 1 ) and r(;Ar(xC_^) =-ct^v/c + o(l). (Ill) 

In the same way, ^ has a unique positive minimum ct^( 1 + v/c)^ at on the interval (0,cr^vA + e), and 

thus will also admit a positive minimum on this interval, at the point wn{xi j^), for N large enough, 
and 


+ Vc)^ + 0(1) and wn{xij^) = (j'^VcG-o{l). (112) 

Therefore, we can find such that liminfjv ^ Moreover, if K,' is a compact included in 

y/c, y/c + e), (HO) also implies that for N large enough, 

inf 1 — a'^CNfNiw) > 0 and inf (j)'j^{w) > 0. (113) 

wGK' w£K' 

which proves that (j)N{JC') C 1R\ supp(/XTv) from [16, Lemma 6]. This shows that it necessarily exists a local 
maximum of (j>M with preimage wn{x 2 j~^) > sup/C'. By fixing two points > tf > <7^(1 + v/c)^ such 
that w{t^),w{t 2 ) G Int {1C'), we easily conclude that 

limsupx)''^ < and liminfcc^^ > (114) 

Af-J-oo ’ AT-J-oo 

which proves A-1. By definition of <2 ) '^{'^ 2 ) ^ + e, which implies of course A-2. 
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